Add local build API for direct filesystem builds on MacOS and Windows #27059

Honny1 · 2025-09-12T13:41:52Z

Adds /libpod/local/build endpoint, client bindings, and path translation utilities to enable container builds from mounted directories to podman machine without tar uploads.

This optimization significantly speeds up build operations when working with remote Podman machines by eliminating redundant file transfers for already-accessible files.

This feature is non-functional on Windows when using the Hyper-V provider. The issue stems from the 9p filesystem protocol, which fails to correctly translate file attributes between the host and the virtual machine. This incompatibility causes certain container build operations to fail. As a result, this functionality has been disabled for Windows environments that utilize Hyper-V.

Fixes: https://issues.redhat.com/browse/RUN-3249
Fixes Build part of #26321

Benchmark

Command: podman build -t test ./test-context
Context Dir content:

$ du -a -h | sort -hr
 10G	.
1.0G	./random-9
1.0G	./random-8
1.0G	./random-7
1.0G	./random-6
1.0G	./random-5
1.0G	./random-4
1.0G	./random-3
1.0G	./random-2
1.0G	./random-10
1.0G	./random-1
4.0K	./Dockerfile

Dockerfile:

$ cat Dockerfile
FROM scratch
COPY random-1 random-1
COPY random-10 random-10

Benchmark Results:

Mac OS

Number of runs: 5
Machine:
- Chip: Apple M3 Pro
- RAM: 36 GB
- SSD: 1 TB Apple SSD
Podman machine configuration:
- CPUs: 4
- Memory: 5.1 GB

Providers

applehv

Before: 67.84 s
After: 9.99 s

libkrun

Before: 70.18 s
After: 12.62 s

Windows

Number of runs: 5
Machine:
- CPU: Intel Core i7-8665U
- RAM: 32 GB
- SSD: 500 GB Samsung 970 Evo Plus
Podman machine configuration:
- CPUs: 8
- Memory: 2048 MB

Providers

WSL

Before: 116.1 s
After: 53.86 s

Hyper-V

DNF

Does this PR introduce a user-facing change?

Added /libpod/local/build endpoint and client bindings to enable direct filesystem builds on macOS and Windows (WSL) without tar uploads, significantly speeding up build operations when working with remote Podman machines. This optimization eliminates redundant file transfers for already-accessible mounted directories.

Updated machine port discovery to iterate through all available machine providers instead of checking only a single provider. This change improves machine detection reliability across different provider configurations.

packit-as-a-service · 2025-09-12T13:55:57Z

[NON-BLOCKING] Packit jobs failed. @containers/packit-build please check. Everyone else, feel free to ignore.

baude · 2025-09-25T19:49:37Z

code LGTM ... if you get asked to fix things by others, adding a little commentary inside some of those functions might help the next person. again only if you have to repush and have time.

flouthoc · 2025-10-01T15:48:54Z

internal/localapi/utils.go

+		result[path] = mapping.RemotePath
+	}
+
+	logrus.Debugf("Translation map: %#v\n", result)


Do we need this ? This message has no context. Translation Map for what.

flouthoc · 2025-10-01T15:50:10Z

pkg/api/handlers/compat/images_build.go

+func validateLocalPath(path string) error {
+	if !filepath.IsAbs(path) {
+		return fmt.Errorf("path %q is not absolute", path)
+	}
+
+	if err := fileutils.Exists(path); err != nil {
+		return err
+	}
+	return nil
+}
+


I think we have this function elsewhere as well, could we do a quick search before adding this again ?

I didn't find a similar function, but I moved it to the local API to be provided to others in the future.

flouthoc · 2025-10-01T15:50:36Z

pkg/api/handlers/compat/images_build.go

+			return nil, utils.GetInternalServerError(fmt.Errorf("additionalbuildcontexts must be in name=value format: %q", url))
+		}
+
+		logrus.Debugf("name: %q, context: %q", name, value)


This message needs more context

flouthoc · 2025-10-01T15:51:33Z

pkg/bindings/images/build.go

+		if isStdIn {
+			stdinFile, err := tempManager.CreateTempFileFromReader(contextDir, "podman-build-stdin-*", os.Stdin)
+			if err != nil {
+				return nil, fmt.Errorf("processing stdin: %w", err)


Suggested change

return nil, fmt.Errorf("processing stdin: %w", err)

return nil, fmt.Errorf("processing containerfile from stdin: %w", err)

Honny1 · 2025-10-02T13:58:49Z

@baude, @flouthoc, I addressed your comments.

Honny1 · 2025-10-06T16:15:35Z

PTAL @Luap99 @mheon

internal/localapi/utils.go

TomSweeneyRedHat · 2025-10-07T21:22:55Z

pkg/bindings/images/build.go

+	}
+
+	for _, c := range containerFiles {
+		// Don not add path to containerfile if it is a URL


Suggested change

// Don not add path to containerfile if it is a URL

// Do not add path to containerfile if it is a URL

Nit

TomSweeneyRedHat · 2025-10-07T21:25:07Z

pkg/bindings/images/build.go

+			remoteContainerfile, ok = translationLocalAPIMap[containerfile]
+			if !ok {
+				if !isInContextDir {
+					return nil, fmt.Errorf("containerfile %q not found in translation map", containerfile)


Suggested change

return nil, fmt.Errorf("containerfile %q not found in translation map", containerfile)

return nil, fmt.Errorf("Containerfile %q not found in translation map", containerfile)

I'll just add this one and it's a nit, but a big nit. Customer facing verbaiage should list this with a capital C. I'll just comment here, but there are other intances.

TomSweeneyRedHat · 2025-10-07T21:32:44Z

LGTM
in general. A few nitty things that could be cleaned up later if needed.

… (only WSL) Adds /libpod/local/build endpoint, client bindings, and path translation utilities to enable container builds from mounted directories to podman machine without tar uploads. This optimization significantly speeds up build operations when working with remote Podman machines by eliminating redundant file transfers for already-accessible files. Fixes: https://issues.redhat.com/browse/RUN-3249 Signed-off-by: Jan Rodák <[email protected]>

Replace single provider.Get() with provider.GetAll() loop to search across all available machine providers when finding machine by port. Signed-off-by: Jan Rodák <[email protected]>

Honny1 · 2025-10-08T08:19:31Z

I fixed the nits and rebased on the latest main.

TomSweeneyRedHat · 2025-10-08T14:14:10Z

LGTM
Tests look to be flaking, I restarted.

flouthoc

LGTM

openshift-ci · 2025-10-08T14:34:56Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: flouthoc, Honny1
Once this PR has been reviewed and has the lgtm label, please assign luap99 for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

TomSweeneyRedHat · 2025-10-08T15:02:47Z

Happy Green test buttons!

Luap99 · 2025-10-09T09:47:12Z

pkg/api/server/register_images.go

+	//    description: |
+	//      Absolute path to the build context directory on the server filesystem.
+	//      This directory must contain all files needed for the build.
+	//  - in: query


This sucks a bit in terms of duplication, people are already forgetting to update swagger docs now having two do two places for each build param is gonna make this worse and cause more drift.

But since I am not really aware of any swagger tricks in that regard I guess we just have to live with that for now.

Luap99 · 2025-10-09T09:59:57Z

pkg/bindings/images/build.go

+	contextRemotePath, found := translationLocalAPIMap[options.ContextDirectory]
+	if !found {
+		return nil, fmt.Errorf("cannot access context directory %q for local build", options.ContextDirectory)
+	}
+	requestParts.Params.Set("localcontextdir", contextRemotePath)


Would it not make more sense to remove translationLocalAPIMap?

I don't see what purpose a normal bindings user who get out of it, one could just set options.ContextDirectory to the right value from the beginning same for the AdditionalBuildContexts.

We could just do all the translations in the BuildOptions already in pkg/domain/infra/tunnel instead and not bother the actual bindings with such details.

Luap99 · 2025-10-09T10:00:24Z

pkg/bindings/images/build.go

+// build contexts. Missing mappings will result in build errors.
+//
+// Returns a BuildReport containing the final image ID and save format.
+func BuildLocal(ctx context.Context, containerFiles []string, options types.BuildOptions, translationLocalAPIMap map[string]string) (*types.BuildReport, error) {


The naming is misleading for external consumer of the bindings, a naive reader could assume this means build locally on the current system not on the server via API.
The comment is clear the code reviewer just seeing the function names would likely assume something else.

Something like BuildFromServerContext might be more clear that the paths must exist on the server not the client.

openshift-ci bot added release-note do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. labels Sep 12, 2025

github-actions bot added the kind/api-change Change to remote API; merits scrutiny label Sep 12, 2025

Honny1 force-pushed the build-local-api branch from 9e81b73 to 46d2fc6 Compare September 12, 2025 13:47

Honny1 force-pushed the build-local-api branch 4 times, most recently from cf65a4c to 25cc198 Compare September 19, 2025 12:01

Honny1 force-pushed the build-local-api branch 4 times, most recently from f4c1c04 to f19d0f3 Compare September 22, 2025 16:37

Honny1 marked this pull request as ready for review September 23, 2025 13:53

openshift-ci bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Sep 23, 2025

jakecorrenti mentioned this pull request Sep 23, 2025

Investigate podman build performance versus vfkit containers/libkrun#414

Open

Honny1 mentioned this pull request Sep 23, 2025

Error running podman build with option --build-context on Hyper-V #23429

Open

flouthoc reviewed Oct 1, 2025

View reviewed changes

Honny1 force-pushed the build-local-api branch from f19d0f3 to aa36c82 Compare October 2, 2025 13:48

Honny1 requested a review from flouthoc October 2, 2025 16:00

TomSweeneyRedHat reviewed Oct 7, 2025

View reviewed changes

internal/localapi/utils.go Show resolved Hide resolved

TomSweeneyRedHat reviewed Oct 7, 2025

View reviewed changes

Honny1 force-pushed the build-local-api branch from aa36c82 to 5a5d58c Compare October 8, 2025 08:15

Honny1 added 2 commits October 8, 2025 10:19

Iterate through all machine providers in FindMachineByPort

092aaeb

Replace single provider.Get() with provider.GetAll() loop to search across all available machine providers when finding machine by port. Signed-off-by: Jan Rodák <[email protected]>

Honny1 force-pushed the build-local-api branch from 5a5d58c to 092aaeb Compare October 8, 2025 08:19

flouthoc approved these changes Oct 8, 2025

View reviewed changes

baude added the 5.7 label Oct 8, 2025

Luap99 reviewed Oct 9, 2025

View reviewed changes

	return nil, fmt.Errorf("processing stdin: %w", err)
	return nil, fmt.Errorf("processing containerfile from stdin: %w", err)

	// Don not add path to containerfile if it is a URL
	// Do not add path to containerfile if it is a URL

	return nil, fmt.Errorf("containerfile %q not found in translation map", containerfile)
	return nil, fmt.Errorf("Containerfile %q not found in translation map", containerfile)

Add local build API for direct filesystem builds on MacOS and Windows #27059

Are you sure you want to change the base?

Add local build API for direct filesystem builds on MacOS and Windows #27059

Conversation

Honny1 commented Sep 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmark

Benchmark Results:

Mac OS

Providers

applehv

libkrun

Windows

Providers

WSL

Hyper-V

Does this PR introduce a user-facing change?

Uh oh!

packit-as-a-service bot commented Sep 12, 2025

Uh oh!

baude commented Sep 25, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Honny1 commented Oct 2, 2025

Uh oh!

Honny1 commented Oct 6, 2025

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

TomSweeneyRedHat commented Oct 7, 2025

Uh oh!

Honny1 commented Oct 8, 2025

Uh oh!

TomSweeneyRedHat commented Oct 8, 2025

Uh oh!

flouthoc left a comment

Choose a reason for hiding this comment

Uh oh!

openshift-ci bot commented Oct 8, 2025

Uh oh!

TomSweeneyRedHat commented Oct 8, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Honny1 commented Sep 12, 2025 •

edited

Loading